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Eugene Livshitz 
Abstract 

We show that Orthogonal Greedy Algorithms (Orthogonal Matching Pursuit) 
provides almost optimal approximation on the first [1/(20M)] steps for M-coherent 
^ I dictionaries. 

O ; 

^ 1 Introduction. 

^ ! In this article we continue the research of convergence of greedy algorithms with regards 

^ [ to dictionaries with small coherence (see |3], [1], [7], P, [2], [6], [5]). The study of approx- 

I imation by incoherent dictionaries was mainly motivated by applications to compressed 

^ sensing. In [3], [7], [1] it was shown that Orthogonal Greedy Algorithm (Orthogonal 

. Matching Pursuit) is effective for signal recovering. In this article we discuss this problem 

^ ! from the point of view of Approximation Theory. 

! Let us recall standard definition of Greedy Algorithms theory. We say that a set V 

' from a Hilbert space if is a dictionary if 

L_j; (j) eV ^ II0II = 1, and span© = H. 

^ . We study dictionaries with small values of coherence 

a^ : 

^. M:= sup \{<P,^)\. (1) 

■ Dictionaries with coherence M are called M-coherent. 

§ ■ Orthogonal Greedy Algorithm (OGA) Set fo ■= f e H, G^^^{f,V) ■= 0. 

T— I . For each m > we inductively find Qm+i € 1^ such that 



\{fm,gm+l) \ = sup \{fm,g)\ 

and define 

Grn+lifiT^) '■= Projspan{gi,...,g^+i)(/), 
fni+1 ■= f — G^f{f, V). 

For a function f E H we define its best m-term approximation 



CiGK,(piGV,l<t<m ' 
i=l 



m 



Following V.N. Temlyakov we call inequalities connecting the error of Greedy approx- 
imation and the best m-term approximation Lebesgue type inequalities. 
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The first Lebesgue type inequality for Greedy Algoritfims was obtained by 
A.C Gilbert, M. Muthukrislinan and J. Strauss in j3] 

Theorem A. For every M-coherent dictionary D and any function f E H the in- 
equality 

\\f-G^''^{f,V)\\ = \\U<8m'/'aM) 

holds for all m, 

1 

1 < m < — 1. 

~ ~ 8V2M 

This estimate was improved by J. Tropp [7] (see also paper [1] of D. L. Donoho, M. Elad 
and V.N. Temlyakov.) 

Theorem B. For every M-coherent dictionary V and any function f & H 

WfmW < (1 + 6m)VV„(/), if l<m< ^ 



3M 



D. L. Donoho, M. Elad and V.N. Temlyakov [2] dramatically improved factor in front 
of a. 

Theorem C. For every M-coherent dictionary T> and any function f E H 
Il/Lmiogmjil < 24a^(/), z/ 1 < m < ^^^^2/3 ■ 

V.N. Temlyakov and P. Zheltov [6j improved the upper border for m and proved two 
new Lebesgue type inequalities. 

Theorem D. For every M-coherent dictionary V and any function f (z H 



||/^L2v^j|| < 3a^(/), z/ m2v^ < 

Theorem E. For every M-coherent dictionary V, any function f & H and any fixed 
(5 > 

The aim of this article is to prove the following result. 
Theorem 1. For every M-coherent dictionary D and any function f & H we have 

\\f-G^^^{f,V)\\ = \\f2m\\<SaUf) 

for all 

1 < m < 



20M 
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2 Preliminary lemmas. 

By conditions of Theorem [1] we have 

M < mM < 1/20. (2) 
We use several standard lemmas. 
Lemma 1. For any n, 1 <n < 2m, and 

n 

h = ^ CiCpi, Ci G M, 0i G V, 

i=l 

we have 

max \ {h,(j)i)\ < max \ci\{l + 2mM), (3) 

l<i<n l<i<n 

max \ {h,(f)i) \ > max |ci|(l - 2mM), (4) 

l<i<n l<i<n 

max \ci\ < max \{h,(j)i)\{l + SmM), (5) 

l<i<n l<i<n 

Proof. Using ([1]) we have for any 1 < i < n 

{h,(j)i) = {ci(j)i, (pi) + ( ^ Cj(j)j,(j)i) < 

l<i<n, 



Similarly 



< Cj + — 1) ( max Icjl ] M < Ci + ( max IcJ ) 2mM. 

' l<i<?i / V l<j<n ' 



(h,(f)i) > Ci — \ max IcJ 2mM. 

' l<j<n ' 



The last two inequalities imply Q and (jl]). It follow form ([2]) that 

(1 - 2mM)(l + 3mM) = 1 + mM - Q{mMf > 1 + mM(l - 0.3) > 1. 
To prove ([5]) we estimate 

max \ci\ < max - 2mM)"^ < max + 3mM). 

l<i<n l<i<n l<i<n 

As corollary we obtain 
Lemma 2. Let n < 2m, h ^ H , (pi E V, 1 < i < n. Suppose that 



□ 



Then 

max |ci| < max + 3mM), 

l<i<n l<j<?i 



Proof. Set 
It's clear that 

{h,(f)i) = {h',(j)i), 1 <i <n. 
Thus the lemma follows from inequality (|5]) for h'. □ 

For > 1 we define 

dn ■= {fn-U9n)- (6) 

Let numbers Xi^n, n > 1, I < i < n, satisfy the equality 

n 

fn = fn-l-^Xi^ngi- (7) 

i=l 

Lemma 3. For any n < 2m we have 

\xi,n\ < M\dn\{l + 3mM), 1 < i < n - 1, (8) 

\Xn,n-dn\<M\dn\il + SmM). (9) 

Proof. By definition of OGA 

{fl,9^)=0, l<t<l, (10) 

and 

/„ = / - GOf\f,V) = f- Proj,p,,(,^,...,,„)(/). 

Hence 

fn = fn-1 - Projspan(gi,...,<;„)(/n-l) = fn-1 " dnQn " Proj^p^^^^g^^ g^-^{fn-l - rfnfi'n)- (H) 

Using and ffTOl) we have for h := /„_i — dnQn and 1 < z < — 1 

< \{fn~l,9i)\ + \dn{9i,9n)\ < M\dn\, 

{h,9n) = 0. 

Suppose that x[ ^, 1 < i < n satisfy 

n 

P^ojspan(3i,...,g„ )(/"-! ~ dn9n) = Projspa^n(gj^ = x'i,n9i- 

1=1 

By Lemma [2] 

\x[J < M\dn\il + 3mM), l<i<n (12) 

It follows from (fTTj) that 



n 



/n /n-1 dn9n ^ ] •^i,n9i ^ ^ 



•^i,n9' 



where = a;^„, l<'^<'n- — 1 and x„^„ = (i„ + This and (IT^ complete the 

proof. □ 
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Lemma 4. For 1 < n < 2m — 1 we have 

\dn+i\ < M„|(1 + 1.25M). 

Proof. By definition of OGA 

\{fn-l,gn+l) \ < \ {fn-l,gn) \ = |4 

Using Lemma El dl]) and we liave 



|(/n,fl'n+l)| < \{fn-l - ^ Xi^ngi, 9n+l)\ < \{fn-l, gn+l) \ + ^ \Xt,n{gi, gn+l)\ < 



i=l i=l 

rt-1 



< \dn\ (1 + (2mM(l + 3mM) + 1)M) < |ci„|(l + 1.25M). 

□ 

Lemma 5. For any 1 < I < n < 2m we have 

\dn\ < \di\exp{2.5mM). 



Proof. Using Lemma H] we write 

, , 2.5mM 



2 5mM \ ^"^ 

\dn\ < \di\{l + L25M)"-' < \di\ ( 1 + ) < \di\ exp(2.5mM). 



□ 



3 Notations. 

By tlie definition of tlie best m-term approximation tliere exist aj G M, ipj & T), 1 < j < m, 
and Vq G H sucli tfiat 

m 

f = fo = ^ ajtpj + 'Vo, {vo, i>j) = 0, 1 < j < m, (13) 
\\vo\\ < 1.01a„(/). (14) 

Set 

L := span(7/>i, . . . , VJ, Pl(-) := Proj J-), (■) := Projix(-), 
Vn := (/n), < n < 2m. 
Let numbers aj^n and < n < 2m, I < j < m satisfy equalities 

m 

fn = Plifn) + Ptifn) = Yl Hn^3 + ^n- (15) 



Then 
Define 



J2hn^J=PLif0-fn). (16) 

o-j,n = o.j — bjn, 1<J<'^, 1<'^< 2m. (17) 



Ti:={zG{l,...,2m}: g, e {^Jj}"^^,} . 

T2 :={!,. ..,2m}\ri, 
^i:={jG{l,...,m}: V^,- G {Un=i} , (18) 
S2:={l,...,m}\S,. (19) 
For numbers Xj^n, 1 < i < n < 2m, from ([7]) and for dn from ([6]) we define 

l<i<n, ieT2 

:= E 

l<n<2m, n£T2 

4 Main lemmas. 

Lemma 6. Let 1 < i < n < 2m, i,n & T2. Then we have 

\{Pt{9n),9^)\<l.lM 

Proof. Let 

m 

i=i 

Since n E T2 and 

9n ^ V^i, I {gn, i^j) I < M, 1 < j < m, 

we get by Lemma [2] tliat 

max \cj\ < M(l + 3mM). 

l<j<m 

Tfierefore we liave 

\{Plign),g.)\ = \{9n-PLi9n),9^)\ < \{9n, 9^)\ + {{PM , 9^)\ < 



< 



M +\{y2^J'^v9i)\< M + m( max \cj\] max \{'^pj,g^)\< 

.7 = 1 



< M + (mM)M(l + 3mM) < I.IM. 

□ 



6 



Lemma 7. Let n G Ti then 



\\Vnf < \\Vn-lf + ^.^DM. 

Proof. Let 

Mn:=tl(7^2n{l,...,n}). 

If T2 n {1, . . . , ra} = then x„ = 0, f„ = = fo and nothing to prove, so we may 

assume that m„ > 1. By Lemma |5] 

Mnl < exp(2.5mM) min \di\. (20) 

l<i<n, ieT2 

On the other hand we have 

( min \di\ \ Un < V • < V £ = D. 

^ ^ l<i<n, «eT2 l<i<2m, jeT2 

Combining with ( 120|) we obtain 

\dr\ < exp(2.5mM) — . (21) 



dlun < exp{5mM)D. (22) 
Applying Lemma [3l ([2]) and ( |2T]) we write 

Xn= ^ |a;j,n| = ^ < M|(i„|(l + 3mM)u„ < 

l<i<n, ieT2 l<i<n-l, ieT2 

< M(l + 3mM) exp(2.5mM)(DM„)^/2 < M(l + 3mM) exp(2.5mM)(D2m)^/2 = 

= (2D)i/=^Mm^/2(l + 3mM) exp(2.5mM) < O-lD^/^m"^/^. (23) 

We have that for any /, 1 < / < m we have 

m m 
\{^aj,n-li^j,1pl)\ = li^aj^n-li^j + Vn-U^l)\ < \{fn-U^l)\ < Mn|- 

i=i 3=1 
Then by Lemma [T] we get 



max |ci,>-i| < max |(N aj^n-itpj^tpi)] ] (1 + 3mM) < |(i„|(l + 3mM). (24) 

l<?<m ' \ l<Z<)Ti ' ' / 

V - - j=l / 

Define 

l<i<n,ier2 l<j<n-l,jeT2 

According the definition of Vn we have 
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Il^nll' = - Pirn' < \K-ir + 2\{Vn-l,Pt{h))\ + ||P^^(/l)|P < 

< + 2|(t;„_i,/i)| + ||/if. (26) 

By definition of OGA {fn-i,gi) = 0,l<i<n — 1, tlierefore using (125|1 and ( !T5|l 

=0, 



<J^|aj-„_i| \{^j,Xi^n9i)\- 

j=l l<i<ra-l, ieT2 

Applying (d]), ([22]), (|24]) and Lemma [3] we obtain 

m 

|(^^n-l,/i)| < ^ |aj>-l| ^ \Xi,n{lpj,g.i)\ < 

j=l l<j<n-l, ieT2 

< \dn\{l + 3mM)M|4|(l + 3mM)unmM < 

< {dlun){l + 3mM)^mM^ < D exp(5mM)(l + SmMfmM^. 



+ 2 Y Xi^nXi,n{9i, gi) < 

l<i<n, i£T2 l<i,l<n, i,leT2, i^l 

< MVnP(l + 3mM)\„ + 2M2|d„|2(l + 3mM)X^ < 
< M2|c/„| V(l + 3mM)2 + MVnpMn(l + 3mM)^4mM < 

< L>M2exp(5mM)(l + 3mM)^(l + 4mM). 

From ([2n]) and <^ it follows that 

ll^'nf < \\Vn-lf + 2\{Vn-l,h)\ + ||/lf < 

< Wvn^if + Z^Mexp(5mM)(l + ?,mMf{2mM + M + 4mM2) < 
< ||t;„_if + DMexp(5mM)(l + 3mM)2(3mM + 4(mM)2) < ||t;„_i f + 0.3DM. 

This estimate together with (!23l) proves the lemma. □ 
Lemma 8. Let n E T2 then 

Xn < 1.15|(i„|, 

Proof. Applying Lemma [3] we have 



J2 \^i,n\<dn+ Yl M\dn\{l + 3mM) < 



Xn 

l<i<n, i£T2 1<«<", «6T2 

< \dn\{l + 2mM(l + 3mM)) < + 3mM) < 1.15|4 
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By Lemma [T] we have 



max |aj^„_i| < (1 + 3mM) max 



l<jr<m 



KKm 



(1 + 3mM) max 

KKm 



(l + 3mM) max \{fn-i,^i)\ < (l + 3mM)|rfJ. (27) 

KKm 



Therefore 

\{Vn-l,gn) - dn\ 



{fn-1 ~ 0'j,n-l4'j, Qn) ~ dn 



{fn-l,gn) -^aj^n-l{lpj,gn) - dn 



< 



3=1 



< 



< ( max |aj>-i| I m max \{ipj^gn)\ < {I + 3mM)\dn\mM. (28) 

\l<j<m ' 1^-^-- 



l<j<m 



Set 



-^L ifn-1 Xn^nQn)- 

Using Lemma [31 ([2]) and (128|) we estimate 

2x„,„(t;„_i,5f„) = 2((i„ + (x„,„ - dn)){dn + ((t;n-i,fi'n) - rfn)) > 

> 2(|rf„| - M(l + ?,mM)\dn\){\dn\ - (1 + 3mM)|d„|mM) > 

> 2dl - 2|rf„p(l + 3mM){M + mM) > 2dl - 4|rf„p(l + 3mM)mM, 



Ib^if < (/n-i - aJn^^s-n)!!^ = - x„,„P^ (5(„) || ^ < 

< ||t;„_if -2x„,„(t;„_i,P^^((7„)) +x2^J|P^^(^„)f < 

< \\Vn-l\\^ -'^Xn,n{Vn-l,gn) + xl^n< 

< \\vn^if - 2dl + 4|ci„p(l + 3mM)mM + (|rf„| + M|d„|(l + 3mM)f < 

< \\vn-if - 0.65dl. (29) 

Similar to the proof of Lemma [7] we define 

Ki<n-l,ieT2 

Equahties {fn-i, di) = 0, 1 < i < n — 1 imply that 

{fn-l,h)=0. 
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Using (fT5|) we have 

m m 
= l(/n-l -X^aj,n-l^i - Xn,nPi{.gn),h)\ < ^ | (tt^- /l) | + | X„,„ (P^ (fif^ ),/?,) | < 

j=l l<j<n-l, ieT2 l<j<n-l, jeT2 

Applying ([T]) , ( !27l) , Lemma |3] and Lemma [6] we continue 

m 

\{v'n,h)\<^\aj^n~l\ 5Z + kn.nS^i.n (P^ , ^i) | < 

j=l l<j<n-l, ier2 1<«<?1-1, ieT2 

m 

< max |ai„_i| max |xj„| > > M+ 

l<i<m l<i<n-l, ier2 ' -^-^ 

j=l l<i<n-l, ieT2 

+ + M(l + 3mM)) max |xi,„| I.IM < 

l<i<n— 1, igr2 ' -^-^ 

l<i<n-l, ieT2 

< dl{l + SmMfnmM^ + + M(l + 3mM))(l + 3mM)nM^ < 

< d^(i.i5)22(mM)2 + l.lci^(;i + mM(1.15))(1.15)2mM2 < O.OUdl, (30) 



l<i<n-l, ieT2 l<j,Z<n-l, i,leT2, if^l 

< M^\dn\^{l + SmMfn + 2M^\dn\'^{l + SmMfn'^M < 

< dl{l + 3mMfM{2Mm + 8{Mmf) < O.OOSrf^. (31) 

Using definitions of v'^ and h we write 

(n \ /n-l 

fn-1 ~ ^ ] Xi^nQi I = Pl (/n-l ^ Xn^nQn) ~ Pl I ^ ] Xj^nQi 
i=l J \i=l 

= v'n-Pt { XI I = - Plih), 

\l<i<n-l, ieT2 / 

ll^nf = \\V'S - ^Wn^Ptih)) + \\Pmr < Kf + 2|K, h)\ + ||/.f . 

Applying ([29D, ([30]) and ([3l]) we obtain 

ll^nf <||<f + 2|K,P,^(/.))| + ||/.f < 

< \\vn-if - O.Q5dl + 2(0.014rf^) + O.OOSdl < ||w„_if - O.Gdl 

□ 
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Lemma 9. We have 

2m 

n=l 

Proof. Using Cauchy inequality, Lemma |H] and Lemma [7] we get 

Xn< Yl ^-^^Kl < 

l<n<2m, neT2 l<n<2m, neT2 

1/2 

< 1.15 { V dl\ (2mV/^ < l.lD"^w}'\ 



\l<n<2m, neT2 / 



2m 

EX; = Y] Xn+ Y. ^" - ^" + ^" - 

l<n<2m, neTi 

n=l l<n<2m, nSTi l<n<2m, neT2 l<n<2m, n^T2 

□ 

Lemma 10. We have 

D"^ < 1.33a^(/), 

||^^2m|| < ll^^oll- 

Proof. Applying , Lemma [7] and Lemma [H] we write 



2m 

(1.01a„(/))2 > ||t;o||' > ||i;o||' - \\v2m\? = 

n=l 

= Y (ll^n-lf Y (ll^n-lf -||^nf)> 

l<n<2m, n.eTi l<n<2m, neT2 

> tlTi (-0.3DM) + ^ 0.6dl > m{-0.3DM) + 0.6D > 0.58D. 

l<n<2m, neT2 

Hence 

Z}i/2 < 1.01(0.58)-i/V„(/) < 1.33a,„(/). 

□ 

In the next lemma we use definitions f lT6|) . f|T8|) and ( !T9|) . 
Lemma 11. For ani/ 1 < n < 2m and j G 5*2 

< 0.12DV2^-i/2_ 
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Proof. By definition ([7j) 

n I n / n 

fn- fo = ^Yl ^^^AYI 

1=1 i=l i=l \ l=i 

l<i<n, ieTi \ l=i / l<i<n, jeT2 \ l=i 

Assume that numbers bj^^ and l<j<m, l<n< 2m satisfy 

m / / " 

j=l \l<j<n, ieTi \ l=i 

m / / " 

j=l \l<j<n, ieT2 \ /=« 

It follows from ( !T6|l that 

&j,n = &j,n + fej.n, l<j<?^, ^<n< 2m. (32) 

It's clear that 



and therefore 

hj^n = 0, j G ^2, 1 < ^ < 2m. (33) 

Set 

l<i<n, ieT2 \ l=i 

By Lemma [9] we estimate for each j, I < j < m, 

l<j<n, ieT2 \ /=i / 

n n 2m 

i=i i<i<i, ieT2 1=1 1=1 

According Lemma [2] we have for 1 < j < m and 1 < n < 2m 

\bj,n\ < 1D^I'^m}l'^M{l + 3mM). (34) 
Combining ([2]), (ESD, dMD and (El we obtain 

|&j>| < 1D^I^'m}l'^M{\ + 3mM) < 0.12P)^/2m"^/^ J G ^2, 1 < n < 2m. 

□ 
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5 Proof of Theorem [T]. 

First we estimate |aj,2m|5 j ^ 'S'2. For each 1 < n < 2m by Lemma [1] and Lemma [TT] we 
have 

\dn\ > max \{fn-i,i'j)\ = max \{PL{fn-i),i'j)\ > (1 - 2mM) max |aj,„_i| > 

l<j<m 

> (1 — 2mM) max |aj,„_i| > (1 — 2mM) max |aj — &j>-i| > 

ieS'2 ' jeS'z 



> (1 - 2mM) l^max |aj| - {).l2D^''^m~^'^^ . 



Since {jTi < "ni and jjT2 > m we get 
D = 



^ > m ("(1 - 2mM) ("maxlaj] - 0.12D^/2^-i/2'\ 



l<i<2m, ier2 



Hence 



^max |aj| - 0.12L)^/2m"^/^^ < D^/2m^^/2(l + 3mM), 

max|a,| < LISD^/^^-^/^ + 0.12D^/2^-^/2 = 1.21 D^'^m^^''^ . 
Then by (fT7|) and Lemma [11] for any j G S'2 we obtain 

K2m\ = \a, - b,,2m\ < l.21D^'^m-"^ + 0.12^1/2^-1/2 < 1.4/)l/2^-l/2_ 
We use well known inequality (see, for example, Lemma 2.1 from [2]) 



5^ «i,2mV'j 



< I ^ a^^ g^ j (1 + mM) < m {lAD^I'^m~^^'^f (1.05) < 2.06^. 

\ie52 / 



Using the definition of OGA, (fTSjl . Lemma [TO] and ([T^ we estimate 



2m 



2m 



2m 



cj, l<i<2m 



min II /2m - CiQi II = min || a^- 2mV'i + W2m - Ci^f^ || < 

^<^i<9.m. ' ' a, l<i<2m ' ' ' ' 

i=\ j=l i=l 

m 2m 
< min II ttj- 2m^i - CiQiW + \\v2rn\\ < 

+ ll^oll < 



< min II V'«i,2m^j - y^Q?/^/|| + ||t;o|| < 



< (2.06/^)1/2 + 1.01a„(/) < (2.06)i/2l.33a™(/) + 1.01a^(/) < 3a™(/). 



This completes the proof. □ 

The author is grateful professor V.N. Temlyakov and professor S.V. Konyagin for 
useful discussions. 
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